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1. Introduction 



The goal of this paper is to prove some combinatorial results about a formula 
for quiver varieties given in ||. 

Let X be a non-singular complex variety and Eo —* E\ — > E% — > • ■ • — > -En a 
sequence of vector bundles and bundle maps over X . A set of rank conditions for 
this sequence is a collection of non- negative integers r = (jy) for < i < j < n. 
This data defines a degeneracy locus in X, 

Q r (E.) = {xeX\ rank^z) -> < r y Vi < j} . 

Let 7"ij denote the rank of the bundle Ei. We will demand that the rank 
conditions can occur, i.e. that there exists a sequence of vector spaces and lin- 
ear maps Vq — > Vi — > • • • — > V n so that dim(l^) = r« and rank(Vi — ► Vj) = 
rjj. This is equivalent to the conditions < min(ri rj+ij) for i < j, and 

r u _ r *,i-i _ r Hij + r i+ij-i > for j - i > 2. 

Given two vector bundles E and F on X and a partition A, we let s\(F — E) 
denote the super-symmetric Schur polynomial in the Chern roots of these bundles. 
By definition this is the determinant of the matrix whose (i, j) th entry is the coef- 
ficient of the term of degree Xi + j — i in the formal power series expansion of the 
quotient of total Chern polynomials Ct(E x/ )/ct(F v ). 

The expected (and maximal) codimension for the locus Q r (E.) in X is 

d{r) = - rij) ■ (r l+1J - r y ) . 

i<j 

The main result of [| gives a formula for the cohomology class of Q r (E.) when it 
has this codimension: 

Pr(E.)] = c ^ r ) s Mi (-Ei - E o) ■ ■ ■ s^ n {E n - £„_i) . 

Here the sum is over sequences of partitions /1 = (pi, . . . ,/x n ); the coefficients c M (r) 
are certain integers given by an explicit combinatorial algorithm which is described 
in Section ||. These coefficients are known to generalize Littlewood-Richardson 
coefficients as well as the coefficients appearing in Stanley symmetric functions S , 
|2| . The formula specializes to give new expressions for all known types of Schubert 
polynomials M. 

There is no immediate geometric reason for the products of Schur polynomials 
appearing in the formula. However, it is even more surprising that the coefficients 
Cfj,(r) all seem to be non-negative. Attempts to prove this has led to a conjecture 
saying that these coefficients count the number of different sequences of tableaux 
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satisfying certain conditions Qj. These sequences are called factor sequences and 



The main result in this paper is a proof of this conjecture in some special cases 
which include all situations where the sequence E, has up to four bundles. We will 
also show that the conjecture follows from a stronger but simpler conjecture, for 
which substantial computational verification has been obtained. For both of these 
results, a sign-reversing involution on pairs of tableaux constructed by S. Fomin 
plays a fundamental role. 

In Section [2] we will explain the algorithm for computing the coefficients c M (r), 
as well as the conjectured formula for these coefficients. In Section || we will prove 
a useful criterion for recognizing factor sequences. Section ^ gives an account of 
Fomin's involution, which in Section [5] is used to formulate the stronger conjecture 
mentioned above. Finally, Section ^ contains a proof of this stronger conjecture in 
special cases. 

The work described in this paper can be viewed as a continuation of a joint 
geometric project with W. Fulton, which resulted in the quiver formula described in 
(3). We would like to thank him for introducing us to the subject of degeneracy loci 
during this very pleasant collaboration, and also for numerous suggestions, ideas, 
comments, etc. during the work on this paper. We are also extremely grateful 
to S. Fomin who provided the vital involution mentioned above, and who also 
collaborated with us in the attempts to prove the conjecture. 



This section explains the algorithm for computing the coefficients c M (r) as well as 
the conjecture for these coefficients. We will first explain this in the ordinary case 
described in the introduction. Then we will extend the notions to a more general 
situation, which for many purposes is easier to work with. 

We will need some notation. Let A = Z[h%, I12, ■ ■ •] be the ring of symmetric 
functions. The variable hi may be identified with the complete symmetric function 
of degree i. If / = (ai, Gt2, ■ • ■ , a p ) is a sequence of integers, define the Schur function 
si G A to be the determinant of the p x p matrix whose (i, j) th entry is h ai+ j_i: 



(Here one sets ho = 1 and h- q = for q > 0.) A Schur function is always equal to 
either zero or plus or minus a Schur function s\ for a partition A. This follows from 
interchanging the rows of the matrix defining s/. Furthermore, the Schur functions 
given by partitions form a basis for the ring of symmetric functions || , || . 

We will give the algorithm for computing the coefficients c M (r) by constructing 
an element P r in the n th tensor power of the ring of symmetric functions A®", such 
that 



are defined in Section 



I 



2. Description of the algorithm 



si — det(/i 0i +j-i)i<»,j<p • 
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It is convenient to arrange the rank conditions in a rank diagram: 
Eq — > Ei — * E 2 — > • • • — » E n 



''02 



''12 



?~22 



?Vi,-2,n 



In this diagram we replace each small triangle of numbers 



r i,j-i 



by a rectangle iZy with ^i+i.j — r^ rows and f» i_i — columns. 

-Rij = r 2+l J — r *i 

These rectangles are then arranged in a rectangle diagram: 

Rqi R12 ■ ■ ■ Rn-l,n 

R02 ■ ■ ■ Rn-2,n 



Ran 

It turns out that the information carried by the rank conditions is very well rep- 
resented in this diagram. First, the expected codimension d(r) for the locus Q r (E,) 
is equal to the total number of boxes in the rectangle diagram. Furthermore, the 
condition that the rank conditions can occur is equivalent to saying that the rect- 
angles get narrower when one travels south-west, while they get shorter when one 
travels south-east. Finally, the element P r depends only on the rectangle diagram. 

We will define P r G A®" by induction on n. When n = 1 (corresponding to 
a sequence of two vector bundles), the rectangle diagram has only one rectangle 
R = Rqi . In this case we set 

Pr = SR G A^ 1 

where R is identified with the partition for which it is the Young diagram. This 
case recovers the Giambelli-Thom-Porteous formula. 

If n > 2 we let f denote the bottom n rows of the rank diagram. Then f is a 
valid set of rank conditions, so by induction we can assume that 



(1) 



Pr = 



C M ( r ) S Ml 



is a well defined element of . 
basis element s 



L . Now P r is obtained from P f by replacing each 



mi 

n-l 



s fln _ 1 in (Q) with the sum 



.o-n-l \i=l 
,T„_i 




Ri-l,i &i 

n-i ,— r 



T n -l r 



This sum is over all partitions <j\, . . . , <r n _i and n, . . . , t„_i such that Oi has fewer 
rows than Ri. 



■l.i 



and each Littlewood-Richardson coefficient c^* T . is non-zero. A 
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diagram consisting of a rectangle Ri—ij with (the Young diagram of) a partition 
CTj attached to its right side, and 7j_i attached beneath should be interpreted as 
the sequence of integers giving the number of boxes in each row of this diagram. 

It can happen that the rectangle Ri—i,i is empty, since the number of rows or 
columns can be zero. If the number of rows is zero, then cr, is required to be empty, 
and the diagram is the Young diagram of Tj_i. If the number of columns is zero, 
then the algorithm requires that the length of Oi is at most equal to the number 
of rows ru — Ti-i^ of Ri—i t i, and the diagram consists of er s ; in the top Tn — Ti—i^ 
rows and Tj_i below this, possibly with some zero-length rows in between. 

Next we will describe the conjectured formula for the coefficients c M (r). We will 
need the notions of (semistandard) Young tableaux and multiplication of tableaux. 
In particular we shall make use of the row and column bumping algorithms for 
tableau multiplication. For this and more, see for example ||. 

A tableau diagram for a set of rank conditions is a filling of all the boxes in 
the corresponding rectangle diagram with integers, such that each rectangle Rij 
becomes a tableau Ty. Furthermore, it is required that the entries of each tableau 
Tij are strictly larger than the entries in tableaux above Ty in the diagram, within 
45 degree angles. These are the tableaux Tki with i < k < I < j and (k, I) ^ (i,j)- 

A factor sequence for a tableau diagram with n rows is a sequence of tableaux 
(Wi, . . . , W n ), which is obtained as follows: If n = 1 then the only factor sequence 
is the sequence (Toi) containing the only tableau in the diagram. When n > 2, a 
factor sequence is obtained by first constructing a factor sequence {U\, . . . , U n -i) for 
the bottom n — 1 rows of the tableau diagram, and choosing arbitrary factorizations 
of the tableaux in this sequence: 

' . I',- (I. 

Then the sequence 

(W 1 ,...,W n ) = (T i ■ Pi , Qi ■ Ti2 • P 2 , • • • , Qn-i ■ T n _i, n ) 

is the factor sequence for the whole tableau diagram. The conjecture from Q, 
which is the theme of this paper, can now be stated as follows: 

Conjecture 1. The coefficient c M (r) is equal to the number of different factor se- 
quences (Wi, . . . , W n ) for any fixed tableau diagram for the rank conditions r, such 
that Wi has shape [ii for each i. 

This conjecture first of all implies that the coefficients c u (r) are non-negative and 
that they are independent of the side lengths of empty rectangles in the rectangle 
diagram. In addition it implies that the number of factor sequences does only 
depend on the rectangle diagram and not on the choice of a filling of its boxes with 
integers. 

Example 1. Suppose we are given a sequence of four vector bundles and the fol- 
lowing rank conditions: 

Eq — > Ei — > Ei — > E 3 

14 3 3 

1 2 2 

1 1 
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These rank conditions then give the following rectangle diagram: 

□ 

□ 

From the bottom row of this diagram we get 

Pf = s n . 

Then using the algorithm we obtain 

s n ^ s _ 

and 

P r = «□ <g s | | (g + (g) + 1(g) S y | C 

1 (g s^n ®sn + 1 ® si i i ®«n + 1 ® ® sp . 



Thus the formula for the cohomology class of $l r (E.) has six terms. Now, one 
possible tableau diagram for the given rank conditions is the following: 

LH 



This diagram has the following six factor sequences: 



(a, aaa , 3) . (a, aa, a) . (0. ar^, a) . 



1 1 1 a a 2 

(0, a^, ta) , (0, aaa , a) , (0, aa, 1 



Since only the rectangle diagram matters for the formula, we will often depict a 
rank diagram simply as a triangle of dots in place of a triangle of numbers. This is 
especially convenient when working with paths through the rank diagram, which we 
shall do shortly. Such a diagram will often be decorated with the rectangles from the 
rectangle diagram, or by the tableaux from a tableau diagram. When this is done, 
each rectangle or tableau is put in the middle of the triangle of dots representing 
the numbers that produced the rectangle. In this way the rank conditions used in 
the above example would be represented by the diagram: 



□ 



□ 



□ 
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We will now introduce a generalization of the formula P r . Define a path through 
the rank diagram to be a union of line segments between neighboring rank con- 
ditions, which form a continuous path from roo to r nn such that any vertical line 
intersects this path at most once. 




The length of a path is the number of contained line segments (which is between 
n and 2n). Given a path 7 of length £, we will define an element P 7 E h.® 1 . It is 
convenient to identify the basis elements of S® 1 with labelings of the line segments 
of 7 with partitions. More generally, if I\, . . . , I( are sequences of integers, we will 
identify the labeling of the line segments in 7 by these sequences, left to right, with 
the element Si 1 ® • • • <g> S[ e E A® £ . All basis elements occurring in P 7 will label 
line segments on the side of the rank diagram with the empty partition. If 7 is the 
highest path, going horizontally from roo to r nn , then P 7 is equal to P r . 

We define P 7 inductively as follows. If 7 is the lowest possible path, going from 
roo to ro n to r„„, then we set P 7 = 1 ® 1 <8> • • • <g> 1 E A® 2 ™. In other words P 7 
is equal to the single basis element which assigns the empty partition to each line 
segment. If 7 is any other path, then we can find a path 7' which is equal to 7, 
except it goes lower at one place, in one of the following ways: 



7 7 



Case 1: 



Case 2: 





By induction we may assume that Py is well defined. 

If we are in Case 1 we now obtain P 7 from P 7 / by replacing each basis element 



occurring in P 7 < with the sum 



E 



ar 




For Case 2, let R be the rectangle associated to the triangle where 7 and 7' differ. 
Then P 7 is obtained from Py by replacing each basis element 
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occurring in P 7 < with zero if a has more rows than R, and otherwise with the 
element: 




An easy induction shows that this definition is independent of the choice of 7'. 
The element P 7 has geometric meaning similar to that of P r . It describes the 
cohomology class of a degeneracy locus fi r (7) defined in j|. 

If we are given a tableau diagram, the notion of a factor sequence can also be 
extended to paths. Any factor sequence for a path 7 will contain one tableau for 
each line segment in 7. As with elements of A® , we will often regard such a 
sequence as a labeling of the line segments in 7 with tableaux. 

If 7 is the lowest path from roo to ro„ to r nn then the only factor sequence 
is the sequence (0, . . . , 0) which assigns the empty tableau to each line segment. 
Otherwise we can find a lower path 7' as in Case 1 or Case 2 above. In order to 
obtain a factor sequence for 7 we must first construct one for 7'. 

If we are in Case 1, let (. . . , W, . . . ) be a factor sequence for 7' such that W is the 
label of the displayed line segment, and let W = P ■ Q be an arbitrary factorization 
of W. Then the sequence (. . . , P, Q, . . . ) is a factor sequence for 7. For Case 2, 
let T be the tableau corresponding to the rectangle R. If (. . . , Q, P, . . . ) is a factor 
sequence for 7' with Q and P the tableaux assigned to the displayed line segments, 
then (. . . , Q ■ T ■ P, . . . ) is a factor sequence for 7. 

Finally we define coefficients £^(7) S Z by the expression 

P i = c fW s mi ® ■ • ■ ® s Mf G A® £ 

where £ is the length of 7. Conjecture [l] then has the following generalization: 

Conjecture 1A. The coefficient 0^(7) is equal to the number of different factor 
sequence (Wi, . . . , We) for the path 7, such that Wi has shape //, /or each i. 

3. A CRITERION FOR FACTOR SEQUENCES 

In this section we will prove a simple criterion for recognizing factor sequences. 
As in the previous section we will start by discussing ordinary factor sequences. 

Let {Tij} be a tableau diagram and let (Wi, . . . , W n ) be a sequence of tableaux. 
At first glance it would appear that to check if this sequence is a factor sequence, 
we would have to find all factor sequences (Uj, . . . , {/„_ 1) for the bottom n — 1 
rows of the tableau diagram, as well as all factorizations Ui = Pi ■ Qi, to see if our 
sequence (W\, . . . , W n ) is obtained from any of these, i.e. W, = Qj_i • Ti-\ t i ■ Pi 
for all i. Equivalently we could find all factorizations of each Wi into three factors 
Wi = Qi-i-Ti^i.i-Pi (with Q = P„ = 0), and check if (Pi-Qi,. . . ,P„-i -Q n -i) is 
a factor sequence for any of these choices. The criterion for factor sequences allows 
us to check this for just one factorization of each Wj. 

Notice that if the sequence (Wi, . . . , W n ) is a factor sequence, obtained from 
an inductive factor sequence (Ui, . . . , U n -i) as above, then the conditions on the 
filling of a tableau diagram imply that the entries of each tableau Ti_i.i are strictly 
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smaller than the entries of Qi-i and Pj. This implies that Wj = Qi-i ■ Tj-i^ • Pi 
contains the rectangular tableau Ij-i^ in its upper-left corner. 



We shall therefore investigate ways to factor a tableau into three pieces, one of 
which is a contained rectangular tableau. 

A quick way to factor any tableau is by cutting it along a horizontal or vertical 
line. Let T be a tableau and a > an integer. Let U the top a rows of T, and D 
the rest of T. Then T = D ■ U . We will call this factorization the horizontal cut 
through T after the a th row. Vertical cuts are defined similarly. 



Lemma 1. Let T = P ■ Q be any factorization of T and let a be the number of 
rows in Q. The following are equivalent: 

(i) T = P ■ Q is a horizontal cut. 

(ii) The i th row of T has the same number of boxes as the i th row of Q for 1 < 



(iii) Whenever the top row of P has a box in column j > 1, the a th row of Q has 
a strictly smaller box in this column (unless a = 0). 

Similarly, if P has b columns, then T = P-Q is a vertical cut iff the first b columns 
of T and P have the same heights, iff the boxes in the last column of P are smaller 
than or equal to the boxes in similar positions in the first column of Q. 

Proof. It is clear that (i) implies (ii) and (iii). If (iii) is true then P and Q fit 
together to form a tableau with Q in the top a rows and P below. By taking a 
horizontal cut through this tableau, we see that it must be the product of P and 
Q. But then it is equal to T and (i) follows. Finally, suppose (ii) is true. When the 
boxes of P are column bumped into Q to form the product T, all of these boxes 
must then stay below the a th row. This process therefore reconstructs P below Q 
and (i) follows. The statements about vertical cuts are proved similarly. □ 

Now let W be any tableau whose shape contains a rectangle (b) a with a rows 
and b columns. We define the canonical factorization of W with respect to the 
rectangle (b) a to be the one obtained by first taking a horizontal cut through W 
after the a th row, and then a vertical cut through the top part of W after the 6 th 
column. 






i < a. 




Note that this definition depends on a, even when b is zero and the rectangle (b) a 
is empty. When the product of three tableau Q, T, and P looks like in this picture, 
we shall say that the pair of tableaux (Q, P) fits around the rectangular tableau T. 



ON A CONJECTURED FORMULA FOR QUIVER VARIETIES 



9 



More generally, let Qo be the part of W below T, P$ the part of W to the right 
of T, and let Z be the remaining part between Qq and Pp. 



We define a simple factorization of W with respect to the rectangle (b) a to be 
any factorization W — Q ■ T ■ P, such that Q — Qo ■ Q and P = P ■ P for some 
factorization Z = Q ■ P. 

Note that if Z = Q ■ P is any factorization of Z and if we put Q = Qo ■ Q and 
P = P-Po, then Q-T-P = W. This follows because P = P-Pq must be a horizontal 
cut through P, and therefore T ■ P = P ■ T ■ Pq. In fact, given arbitrary tableaux 
Q and P one can show that Q ■ T ■ P = W if and only if Q ■ P = Z , but we shall 
not need this here. 

We are now ready to formulate the criterion for factor sequences. Let {Rij} be 
the rectangles corresponding to the tableau diagram {Ty}. If (W\, . . . ,W„) is a 
factor sequence, a simple factorization of any Wi will always be with respect to the 
relevant rectangle Ri—i,i from the rectangle diagram. 

Theorem 1. Let (W\, . . . , W n ) be a sequence of tableaux such that each Wi con- 
tains Ti—i t i in its upper-left corner. Let Wi — Qi-\ ■ Ti^i_i ■ Pi be any simple factor- 
ization of Wi with respect to the rectangle Ri^ii. Then (Wi, . . . ,W n ) is a factor 
sequence if and only if Qq and P n are empty tableaux and (Pi • Q\ , . . . , P n -x ■ Qn—i) 
is a factor sequence for the bottom n — 1 rows of the tableau diagram {Tij}. 

We shall derive this result from Proposition below. Since this criterion can 
be applied recursively to the sequence (Pi ■ Q±, . . . , P n -\ • Qn-i), it gives an easy 
algorithm to determine if a sequence (Wi, . . . ,W n ) is a factor sequence. Note 
that the easiest way to produce the simple factorizations is to take the canonical 
factorization of each Wi. When this choice is made, the work required in the 
algorithm essentially consists of n(n — l)/2 tableau multiplications. Note also that 
this criterion makes use of the height of any empty rectangles in the rectangle 
diagram. 

For proving this criterion we need some definitions. Let T be a tableau whose 
shape is the rectangle (b) a with a rows and b columns. We will consider pairs of 
tableaux (X, Y) such that all entries in X and Y are strictly larger than the entries 
of T. For such a pair, let X = Xq ■ X be the vertical cut through X after the 6 th 
column, and let Y = Y ■ Yq be the horizontal cut after row a. 



If (X' , Y') is another pair of tableaux, we will write (X, Y) \= (X' , Y') if either 

1. for some factorization X = M ■ N we have X' = Xq ■ M and Y' = N ■ Y , or 

2. for some factorization Y = M ■ N we have X' = X ■ M and Y' = N -Y . 




T 
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Note that this implies that X' ■ T ■ Y' = X ■ T ■ Y . In in the first case this follows 
because X ■ T — X T-X and X' ■ T = X n -T-Al, and the second case is similar. We 
will let — > denote the transitive closure of the relation |=. This notation depends 
on the choice of T, as well as the numbers a and b if T is empty. 

Lemma 2. Let W be a tableau containing T in its upper-left corner. Suppose that 
the entries of T are smaller than all other entries in W . If W = Q ■ T ■ P is a 
simple factorization of W with respect to the rectangle (b) a , and W — X ■ T ■ Y is 
any factorization, then (X, Y) — > (Q,P). 

Proof. Let X = Xq ■ X be the vertical cut through X after column b, and put 
Y' = X -Y. Then let Y' = Y' • F ' be the horizontal cut through Y' after row a, 
and put X" = Xq ■ Y'. 

We claim that the pair {X" , Y" ') fits around T. Using Lemma |l| and that the 
entries of T are smaller than all other entries, it is enough to prove that the b + j th 
entry in the top row of X" is strictly larger than the j th entry in the bottom row 
of Y" '. This will follow if the b + j th entry in the top row of X" is larger than or 
equal to the j th entry in the top row of Y' . Since X" = Xq ■ Y' and Xq has at most 
b columns, this follows from an easy induction on the number of rows of Y'. 

It follows from the claim that W — X" ■ T ■ Yq is the canonical factorization of 
W, and therefore we have (X, Y) \= {X , Y') (= (X", F ') |= (Q, P) as required. □ 

Notice that if W = X ■ T ■ Y is a simple factorization and (X, Y) \= (X',Y r ), 
then W = X' ■ T ■ Y' must also be a simple factorization. It follows that Lemma |^ 
would be false without the requirement that W — Q ■ T ■ P is simple. 

Lemma 3. Let a > be an integer, and let Y and S be tableaux with product 
A = Y ■ S . Let A = A ■ Aq and Y = Y ■ Y a be the horizontal cuts through A and Y 
after row a, and let Y — M ■ N be any factorization. Then N ■ Yq ■ S — A' ■ Aq for 
some tableau A' and M ■ A' = A. 



Y I 

Y = pJ ; A = Y-S = 

Y 




Proof. The first statement follows from the observation that the bottom rows of Y 
can't influence the top part of Y ■ S, which is a consequence of the row bumping 
algorithm. Lemma |l then shows that the factorization A = (M ■ A') ■ Aq is a 
horizontal cut, so M ■ A' = A as required. □ 



Lemma 4. Let 7 be a path through the rank diagram, and let (. . . , A, B ■ C ', . . . ) be 
a factor sequence for 7 such that the product B ■ C is the label of a down-going line 
segment. Then (. . . , A ■ B, C, . . . ) is also a factor sequence for 7. 

■C 

Proof. We will first consider the case where the line segment corresponding to A 
goes up. Let 7' be the path under 7 that cuts short this line segment and its 
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successor. 




A-B-C 



Then by definition (. . . , A ■ B ■ C, . . . ) is a factor sequence for 7', which means that 
(. . . , A ■ B, C, . . . ) is a factor sequence for 7. In general 7 lies over a path like the 
one above, and the general case follows from this. □ 

Similarly one can prove that if (. . . , A ■ B, C, . . . ) is a factor sequence for a path, 
such that A ■ B is the label of an up-going line segment, then (. . . , A, B ■ C, . . .) is 
also a factor sequence for this path. 

Proposition 1. Let 7 and 7' be paths related as in Case 2 of Section ^, and let 
(. . . , W, . . . ) be a factor sequence for 7 such that W is the label of the displayed 
horizontal line segment. 

W 

If W — Q ■ T ■ P is any simple factorization of W , then {. . . , Q, P, . . . ) is a factor 
sequence for 7'. 

Proof. Since (. . . , W, . . . ) is a factor sequence for 7, there exists a factorization 
W — X ■ T ■ Y such that (. . . , X, Y, . . .) is a factor sequence for 7'. By Lemma |^ 
we have (X, Y) -> (Q, P). It is therefore enough to show that if (X, Y) |= (X' , Y') 
then (. . . . X', Y', . . . ) is a factor sequence for 7'. 

Let a be the number of rows in (the rectangle corresponding to) T, and let 
Y = Y ■ Yo be the horizontal cut through Y after the a th row. We will do the case 
where a factor of Y is moved to X, the other case is proved using a symmetric 
argument. We then have a factorization Y — M ■ N such that X' = X ■ M and 
Y' = N ■ Yq. We can assume that the paths 7 and 7' go down after they meet, and 
that the original factor sequence for 7 is (. . . , W, S, . . . ). 



W 




Put A — Y ■ S. Then (. . . , X, A, . . . ) is a factor sequence for the path with these 
labels in the picture. Now let T' be the rectangular tableau associated to the 
lower triangle, and let A = U ■ T' ■ V be the canonical factorization of A. Since 
this is a simple factorization we may assume by induction that (. . . ,X, U, V, . . .) 
is a factor sequence. Using Lemma § we deduce that N ■ Yo ■ S — U' ■ T' ■ V for 
some tableau U', such that M ■ U' = U. Since (. . . , X, M ■ U' , V, . . . ) is a factor 
sequence, so is (. . . , X ■ M, U',V,...) by Lemma § This means that (. . . , X ■ M, U' ■ 
T' ■ V, . . .) — (. . . , X' , Y' ■ S, . . . ) is a factor sequence, which in turn implies that 
(. . . , X', Y', S, . . . ) is a factor sequence for 7' as required. □ 



The proof of Proposition pi also gives the following: 
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Corollary. Let (. . . , X, Y, . . . ) be a factor sequence for the path 7' in the propo- 
sition. If [X, Y) — » (X', y') then (. . . , X', Y 7 , . . . ) is also a factor sequence for 

i- 

Proof of Theorem The "if" implication follows from the definition. If the se- 
quence (Wi, . . . ,W n ) is a factor sequence, then n applications of Proposition [l] 
shows that (Qoj Pi-, Qu P2, ■ ■ ■ > Qn-ii Pn) IS a factor sequence for the path with 
these labels. 

Wl W 2 W 3 W n 



-1) is a factor 
□ 





It follows that Qo and P n are empty, and (Pi • Q%, . . . , P n -i 
sequence for the bottom n — 1 rows. This proves "only if" . 

4. An involution of Fomin 



Q, 



In this section we will describe a sign-reversing involution on pairs of tableaux 
constructed by Sergey Fomin. The purpose of this involution is to cancel out the 
difference between the coefficients c^(r) produced by the algorithm in Section |[ 
and their conjectured values. 

Fix a positive integer a. If P and Q are tableaux of shapes a and r such that P 
has at most a rows, we let S(^) denote the symmetric function sj G A where / is 
the sequence of integers / = (cri , . . . , a a , T\, T2, . . . ). Let V a be the set of all pairs 
(Q, P) such that 7^ and such that P and Q do not fit together as a tableau 

with P in the top a rows and Q below. This means that the a th row of P must be 
shorter than the top row of Q, or some box in the top row of Q must be smaller 
than or equal to the box in the same position of the a th row of P. For example, if 
a = 2 the following pairs are in P a : 



6 7 



1 3 



2 4 



) and ( _ 



4 


4 


7 


5 


6 











2 


3 


3 


4 


3 


4 


5 


5 



Lemma 5 (Fomin's involution). There exists an involution ofP a with the property 
that if (Q, P) is mapped to (Q', P 1 ) then 

(i) Q' P' = Q P, 

(ii) S{%) = -S(%), and 

(iii) the first column of Q' is equal to the first column of Q. 

Fomin supplied the proof of this lemma in the form of the beautiful algorithm 
described below. While Fomin's original description uses path representations of 
tableaux, we have translated the algorithm into notation that is closer to the rest 
of this paper. 

We will work with diagrams with weakly increasing rows. These will be "Young 
diagrams" for finite sequences of non-negative integers, where all boxes are filled 
with integers so that the rows are weakly increasing. Empty rows are allowed as in 
the following example: 
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A violation for such a diagram to be a tableau is a box in the second row or below, 
such that there is no box directly above it, or the box directly above it is not strictly 
smaller. The above diagram has 4 violations in its second row and 2 in row four. 

If D is a diagram with weakly increasing rows, and if I is the sequence of row 
lengths, we put S(D) = sj G A. Let rect(D) denote the tableau obtained by 
multiplying the rows of D together, from bottom to top. We will identify a pair 
(Q, P) <E P a with the diagram D consisting of P in the top a rows and Q below. 
For this diagram we then have Q ■ P = rect(D) and S(^) = S(D). 

We will start by taking care of the special case where a = 1 and both P and Q 
have at most one row. In this case Lemma ^| without property (iii) is equivalent 
to the identity sg^ = hihk — /i^+i^fc-i in the plactic monoid, which is a special 
case of a result by Lascoux and Schutzenberger [ffoj, (§). The simple proof of this 
result given in [Jjj develops techniques which Fomin used to establish Lemma [5] in 
full generality. 

Lemma 6. Let D be a diagram with two rows and at least one violation in the 
second row. Then there exists a unique diagram D' such that rect(D') = rect(Z?) 
and S(D') = —S(D). Furthermore, D' also has two rows and at least one violation 
in the second row. The leftmost violations of D and D' appear in the same column 
and contain the same number. The parts of D and D' to the left of this column 
agree. 

Proof. Let p and q be the lengths of the top and bottom rows of D. The requirement 
S(D') — —S(D) then implies that D' must have two rows with q — 1 boxes in the 
top row and p + 1 in the bottom row. Now it follows from the Pieri formula §, §2.2] 
that the product rect(D) of the rows in D has at most two rows. Furthermore, 
since D contains a violation, the second row of rect(Z?) has at most q — 1 boxes. 
Using the Pieri formula again, this implies that there is exactly one way to factorize 
rect(D) into a row of length p + 1 times another of length q — 1. This establishes 
the existence and uniqueness of D' . 

Explicitly, one may use the inverse row bumping algorithm to obtain this factor- 
ization of rect(-D). This is done by bumping out a horizontal strip of q — 1 boxes 
which includes all boxes in the second row, working from right to left. 

Let x be the leftmost violation of D, where D has the form: 



D = 



A 


E | 


B 


x\ F 



Suppose the parts A and B each contain t boxes. Now form the product F ■ E and 
let Cj and dj be the boxes of this product as in the picture: 



Cl 


C2 


C3 M • • • \ c k 


di 


do 


■■■ \di\ 



Since a; is a violation in D, it must be smaller than all boxes in E and F. Therefore 
we have 



X 


Cl 


C2 C 3 C 4 


• • ■ Cfc 


di 


d 2 




M 



Now since each dj > Cj it follows that if a horizontal strip of length q — t — 1 is 
bumped off this tableau, x will remain where it is. In other words we can factor 
x ■ F ■ E into x ■ F' ■ E' such that x ■ F' and E' are rows of lengths p — t + 1 and 
q — t — 1 respectively. Since the entries of A and B are no larger than x, the products 
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B ■ x ■ F' and A ■ E' are rows of lengths p + 1 and q—1. But the product of these 
rows is rect(L>), so they must be the rows of D' by the uniqueness. This proves 
that D' has the stated properties. □ 

Notice that the uniqueness also implies that the transformation of diagrams 
described in the lemma is inverse to itself, i.e. an involution. 

Now suppose D is any diagram with weakly increasing rows. Then Lemma |^ 
can be applied to any subdiagram of two consecutive rows, such that the second of 
these rows contains a violation. If this subdiagram is replaced by the new two-row 
diagram given by the lemma, we arrive at a diagram D' satisfying S(D') — —S(D) 
and rect(D') = rect(D). We will call this an exchange operation between the two 
rows of D. 

We shall need an ordering on the violations in a diagram. Here the smallest 
of two violations is the south-west most one. If the two violations are equally far 
south-west, then the north-west most one is smaller. In other words, a violation in 
row i and column j is smaller than another in row i' and column j' iff j — i < j' — i', 
or j — i = j' — %' and i < i' . 

Notice that when an exchange operation between two rows is carried out, viola- 
tions may appear or disappear in these two rows as well as in the row below them. 
However, the properties given in Lemma ^| imply that all of the changed violations 
will be larger than the left-most violation in the second of the rows exchanged. It 
follows that the minimal violation in a diagram will remain constant if any (se- 
quence of) exchange operations is carried out. Similarly, all boxes south-west of 
the minimal violation will remain fixed. 

Proof of Lemma\^. Given a pair (Q,P) G Pa, let T>q p be the finite set of all non- 
tableau diagrams D with weakly increasing rows, such that rect(Z?) = Q ■ P and 
S(D) = +S(^), and so that the minimal violation in D is in row a + 1. The pair 
(Q,P) is then identified with one of the diagrams in this set. We will describe an 
involution of the set T>q p and another of the complement of P a n T)q,p in Vq p. 
The restriction of Fomin's involution to V a n T>q p is then obtained by applying 
the involution principle of Garsia and Milne |tJ to these involutions. 

The involution of £>q.p simply consists of doing an exchange operation between 
the rows a and a+l of a diagram. This is possible because all diagrams are required 
to have a violation in row a + 1 . 

Now note that a diagram D s 2?q,p is in the complement of V a H P>q,p if and 
only if D has a violation outside the a+ 1 st row. We take the involution of T>Q t p\V a 
to be an exchange operation between the row of the minimal violation outside row 
a + 1, and the row above this violation. This is indeed an involution since the 
minimal violation outside row a + 1 stays the same. 

These involutions now combine to give an involution of V a R £*q,p by the invo- 
lution principle. To carry it out, start by forming the diagram with P in the top a 
rows and Q below it. Then do an exchange operation between row a and row a+l. 
If all violations in the resulting diagram are in row a + l we are done. P' is then 
the top a rows of this diagram and Q' is the rest. Otherwise we continue by doing 
an exchange operation between the row of the minimal violation outside row a+l 
and the row above it, followed by another exchange operation between row a and 
row a+l. We continue in this way until all violations are in row a + l. 
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Finally, the properties of P' and Q' follow from the properties of exchange oper- 
ations. In particular, the requirement S(^r) = —S(^) follows because we always 
carry out an odd number of exchange operations. □ 



Example 2. The pair (P, Q) -- 
quence of exchange operations: 





5 | 6 | 7 


1 


3 


7 8 


i 




2 


4 





) in V2 gives the following se- 



1 


3 


7 8 |j_ 


3 


7 


8 


1 


3 




1 


3 




2 


4 


2 


4 


7 




2 


4 


7 


7 8 


2 


4 




3 


5 


6 1 7 1 ~» | 3 


5 


6 




3 


5 


6 




3 


5 


6 7 7 8 


j4 


\a 








J_ 








4 





This pair therefore corresponds to (P',Q') 
hit ion. 



& 



j) by Fomin's invo- 



There are examples of pairs (Q, P) for which the set P a n £>q,p has more than 
two diagrams, all with the same first column. This means that the involution 
constructed above is not the only one that satisfies the conditions of Lemma ^[ One 
way to produce different involutions is to use another ordering among violations. 
The only property of the order that we have used is that when an exchange operation 
is performed, any appearing and disappearing violations must be larger than the 
leftmost violation in the second of the rows being exchanged. For example, given 
any irrational parameter £ £ (0, 1), we obtain a new order by letting a violation in 
position be smaller than another in position (i' if and only if j — £i < 

5. The stronger conjecture 

In this section we will present a simple conjecture which implies Conjecture 1A. 
Let 7 be a path through the rank diagram which at some triangle has an angle 
pointing down: 




Let T be the rectangular tableau associated to this triangle, and suppose the cor- 
responding rectangle has a rows and b columns. 

If X and Y are tableaux whose entries are strictly larger than the entries of T, 
and if Y has at most a rows, we will let 



T\Y 
~X~ 



T 
X 



denote the diagram consisting of T ■ Y in the top a rows and X below. The sequence 
of row lengths of this diagram then gives an element S(^-) in the ring of symmetric 
functions A. Note that (X, Y) fits around T if and only if the diagram is a 
tableau. 

Suppose that (X,Y) does not fit around T and S(^f-) is non-zero. Let X = 
Xq ■ X be the vertical cut through X after the 6 th column. Then (X, Y) is an 
element of the set V a defined in the previous section. Let (X',Y') be the result 
of applying Fomin's involution to this pair, and set X' = Xq ■ X'. Since the first 
columns of X and X' agree, X' consists of X with X' attached to its right side by 



16 



ANDERS SKOVSTED BUCH 



ST I y ' T I V 

. It follows that S( j^, ) = — (Note that one could also get from 

(X, Y) to (X', Y') by applying Fomin's involution to the pair (X, T ■ Y).) 

Conjecture 2. Let (. . . , X,Y, . . .) be a factor sequence for 7 with X and Y the 
labels of the displayed line segments, such that Y has at most a rows. Suppose 
(X,Y) does not fit around T and S(^-) ^ 0. If X' and Y' are obtained from X 
and Y by applying Fomin's involution as described above, then (. . . , X' , Y', . . . ) is 
also a factor sequence for 7. 

If we fix the location of the down-pointing angle of 7 (i.e. the location of T in the 
tableau diagram), then the strongest case of this conjecture is when the rest of 7 
goes as low as possible. If Conjecture || is true for all locations of the down-pointing 
angle, then the conjectured formula for the coefficients 0^(7) is correct. 

Theorem 2. Conjecture 1A follows from Conjecture 

Proof. If W\, . . . , We are diagrams with weakly increasing rows, e.g. tableaux, we 
will write S(Wi, ...,W t ) = S(Wi) ® • •• <8> S(W e ) G A®^. With this notation we 
must prove that if 7 is a path of length £, then 

(2) R r =Y,SQVi,...,W i ) 

(Wi) 

where the sum is over all factor sequences (Wi) for 7. 

Let 7' be a path under 7 as in Case 1 or Case 2 of Section |[ By induction we 
can assume that Conjecture 1A is true for 7', i.e. 



(3) Py =^S(Ul,...,U t ,) 

where this sum is over the factor sequences for 7'. We must prove that the right 
hand side of @ is obtained by replacing each basis element of (||) in the way 
prescribed by the definition of P 7 . If we are in Case 1 then this follows from the 
Littlewood- Richardson rule || §5.1]: If U is a tableau of shape fi and a and r are 
partitions, then there are c£J T ways to factor U into a product U = P ■ Q such that 
P has shape a and Q has shape r. 

Assume we are in Case 2. By induction we then have Py = ■ ■ 7 AT, Y, . . . ) 

where the sum is over all factor sequences (. . . , X,Y, . . .) for 7'; X and Y are the 
labels of the two line segments where 7' is lower than 7. Let T be the rectangu- 
lar tableau of the corresponding triangle, and let a be the number of rows in its 
rectangle. Then by definition we get 



(4) P 7 = J2 S(...,^-,...) 

where the sum is over all factor sequences (. . . , X, Y, . . . ) for 7' such that Y has at 
most a rows. 

Now suppose we have a factor sequence (. . . , X, Y, . . . ) such that the diagram 
is a tableau. Then this tableau must be the product X -T-Y, and so (. . . , . . . ) 

is a factor sequence for 7. Thus the term S(. . . , ^p-, ■ ■ • ) matches one of the terms 
of (||) . On the other hand it follows from Proposition [| that every term of (0) is 
matched in this way. 

We conclude from this that the terms in (0) is the subset of the terms in 
which come from factor sequences such that (X, Y) fits around T. We claim that 



T\Y 
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the sum of the remaining terms in (0) is zero. In fact, if (. . . ,X, Y, . . . ) is a factor 
sequence for 7' such that (X, Y) doesn't fit around T and S(^-jsf-) ^ 0, then we may 
apply Fomin's involution in the way described above to get tableaux X' and Y 1 . If 
Conjecture || is true, then the sequence (. . . , X' , Y', . . . ) is also a factor sequence, 

and since £(-^7-) = — S(-^~), the terms of (|4|) given by these two factor sequences 
cancel each other out. □ 

The number of factor sequences for a tableau diagram can be extremely large. 
For this reason it is almost impossible to verify Conjecture [l] or Conjecture 1A by 
computing both sides of their equations. In contrast, instances of Conjecture || can 
be tested easily even on large examples. Given a tableau diagram and a path, one 
can generate a factor sequence for this path by choosing factorizations of tableaux 
by random. Then one can apply Fomin's involution to the sequence, and use the 
criterion of Proposition [l] to check that the result is still a factor sequence. Such 
checks have been carried out repeatedly for each of 500,000 randomly chosen tableau 
diagrams with up to 10 rows of tableaux, without finding any violations of Con- 
jecture ^| Together with the results in the next section, we consider this to be 
convincing evidence for the conjectures. 

6. Proof in a special case 

In this final section we will show that Conjecture |^ is true in certain special cases. 
These cases will be sufficient to prove the conjectured formula for c M (r) when all 
rectangles in and below the fourth row of the rectangle diagram are empty, and 
when no two non-empty rectangles in the third row are neighbors. This covers all 
situations with at most four vector bundles. 

Let 7 be a path through the rank diagram with a down-pointing angle as in the 
previous section. Let R be the rectangle of the corresponding triangle. 




We will describe two cases where Conjecture y can be proved. Both cases require a 
special configuration of the rectangles surrounding R. Suppose R is the rectangle 
Rij in the rectangle diagram. We will say that a different rectangle R' — Rki is 
below R if k <i < j < I. R' is strictly below R\ik<i<j<l. 

Proposition 2. Conjecture |^ is true for 7 if all rectangles strictly below R are 
empty. 




rectangles 



Note that this covers all rectangles on the left and right sides of the rectangle 
diagram. 

Proof. Let T be the tableau corresponding to R, and suppose (. . . , X, Y, . . . ) is a 
factor sequence for 7. Since all tableau on the line going south-west from T in the 
tableau diagram are narrower than T, it follows that also X has fewer columns than 
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T. Similarly Y has fewer rows than T. But this means that (X, Y) fits around T 
and the statement of Conjecture || is trivially true. □ 

In the other situation we shall describe, we allow three non-empty tableaux below 
T as shown in the picture. 

7 

m m 
m 

All other tableaux below T are required to be empty. Let 7 be the higher and 7' 
the lower of the two paths in the diagram. 

Lemma 7. Let (. . . , X, Y, . . . ) be a labeling of the line segments ofj' with tableaux. 
The following are equivalent: 

(1) (. . . ,X,Y, . . .) is a factor sequence for 7'. 

(2) (. . . , X ■ T ■ Y, . . . ) is a factor sequence for 7 and the part of X that is wider 
than T and the part ofY that is taller than T have entries only from C. 

Proof. It is clear that (1) implies (2). For the other implication, put W — X • T -Y 
and let W = X' ■ T ■ Y' be the canonical factorization of W . Then it follows from 
Proposition [l] that (..., X' ,Y' , . . .) is a factor sequence for 7'. Since (X, Y) — » 
(X', Y 1 ) by Lemma 0, we may assume that (X, Y) \= (X' , Y'). 

We will handle the case where a factor of the bottom part of Y is moved to 
X, the other case being symmetric. This means that for some tableau M we have 
X' = X ■ M and Y = M ■ Y'. Since the bottom part of Y has entries only from C, 
this is also true for M. 

We may assume that 7 and 7' go down outside the displayed angle and that our 
factor sequence is (. . . , U, X',Y', V, . . . ). 



W 




Then by definition there exists a factorization C = C{ • C' 2 such that A-C[ = U ■ X' 
and C' 2 -B = Y' -V . Since U ■ X ■ M consists of A with C[ attached on its right side, 
and since all entries of M are strictly larger than the entries of A, it follows that 
U ■ X consists of A with some tableau C\ attached on the right side. Furthermore 
d ■ M = C[ by Lemma 0. 

Put C 2 = M ■ C' 2 . Then we have C x ■ C 2 = C, A ■ C x = U ■ X, and C 2 • B = Y ■ V. 
It follows that (. . . , U, X, Y, V, . . . ) is a factor sequence as required. □ 

Proposition 3. Conjecture is true for the path 7' in Lemma ^. 

Proof. Let (. . . ,X, Y, . . , ) be a factor sequence for 7' which satisfies the conditions 
in Conjecture ^, and let X' and Y' be the tableaux obtained from X and Y using 
Fomin's involution. Since the part of X that is wider than T has entries only from 
C, the same will be true for X' by Lemma ^ (iii). Since Y' has fewer rows than T 
and since (. . . , X' ■ T ■ Y' , ...) = (..., X ■ T ■ Y, • • • ) is a factor sequence for 7, it 
follows from Lemma I that (...,X\Y',...) is a factor sequence for 7'. □ 
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Corollary. Conjecture |j] is true if all rectangles in and below the fourth row of the 
rectangle diagram are empty, and if no two non-empty rectangles in the third row 
are neighbors. 

Proof. When the rectangle diagram satisfy these properties, then all instances of 
Conjecture || follow from either Proposition || or Proposition ||. The corollary 
therefore follows from Theorem ^. □ 

In Section ^| we defined a rectangle diagram to be something you get by replacing 
the small triangles of numbers in a rank diagram with rectangles. However, every- 
thing we have done is still true if one defines a rectangle diagram to be any diagram 
of rectangles, each given by a number of rows and columns, such that the number of 
rows decreases when one moves south-east while the number of columns decreases 
when one moves south-west. This definition is slightly more general because the 
side lengths of the rectangles in a rectangle diagram obtained from rank conditions 
satisfy certain relations. Although we don't know any geometric interpretation of 
the more general rectangle diagrams, they seem to be the natural definition for 
combinatorial purposes. 
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